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ABSTRACT 

This paper is a report of user studies for developing 
learning strategy tool software for children. The prototype software 
demonstrated is designed for children with learning and behavioral 
disabilities. The tools consist of easy-to-use templates for creating 
organizational, memory, and learning approach guides for use in classrooms 
and at home. User studies that were conducted with software prototypes 
included two rounds of expert review, usability testing in a lab setting with 
teachers and parents, a focus group with parents and educators, and "think 
aloud" observations of children using a sample of the tools. Results were 
used in improving the software, preparing the teacher orientation module, and 
designing online supports for users. The most useful dimensions of evaluation 
were found to be direct observation of children using the software and 
messages contributed by a high school student with learning disabilities to 
the Online Focus Group. (Contains 19 references and 3 figures.) (Author) 
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Abstract: This paper is a report of user studies for developing learning strategy 
tool software for children. The prototype software demonstrated is designed for 
children with learning and behavioral disabilities. The tools consist of easy-to-use 
templates for creating organizational, memory, and learning approach guides for use 
in classrooms and at home. User studies that were conducted with software 
prototypes included two rounds of expert review, usability testing in a lab setting 
with teachers and parents, a focus group with parents and educators, and “think- 
aloud” observations of children using a sample of the tools. Results were used in 
improving the software, preparing the teacher orientation module, and designing 
online supports for users. The most useful dimensions of evaluation were found to be 
direct observation of children using the software and messages contributed by a high 
school student with learning disabilities to the Online Focus Group. 



Service to children with behavioral and learning disorders is a high priority in the U.S. (Elam, Rose, & 
Gallup, 1998; Kay, 1999). Currently, federal mandates require that such children be served in the “least 
restrictive environment” by providing appropriate supports and effective strategies to ensure success (IDEA97). 
Common problems faced by these students in regular classrooms are difficulties maintaining attention, 
organizing work, managing time and materials, and responding with accuracy (Lewis & Doorlag, 1999). With 
these deficiencies students are at-risk for increased behavioral problems, failure, and ultimately, removal from 
these classrooms. Research shows, however, that strategy instruction in these skills can help students become 
successful learners and managers of their own behavior (Swanson & Hoskyn, 1998). 

Electronic Performance Support for Children with Learning and Behavior Problems 

Computer-based training and support mechanisms are an innovative approach for helping children gain 
control over personal behaviors. While electronic performance support systems have been used in education and 
industry to support adults in learning new skills “at the right place, right time, right form” (Laffey, 1995), work 
is scant in applying these principles to the development of software tools for children. If the goal of EPSS 
software is appropriate for adults — to provide whatever is necessary to ensure performance and learning at the 
moment of need in a seamless activity — then it offers the same potential for helping children improve 
performance in their learning environments. If effective, EPSS support tools for children improve their 
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independent functioning, reduce teacher time re-focusing their mistakes and energies, and offer lifelong skills 
for success. 

This paper reports the design process followed by the developers in creating a new set of EPSS tools to 
help young children learn and use these school survival strategies. These unique materials include four 
components: (1) strategy software tools for children, (2) an information resource database for teachers, (3) a 
web-based orientation module for teachers, and (4) online resources and discussion lists to support teachers and 
parents (refer to conceptual framework in Figure 1) (Fitzgerald & Koury, 2001-2002), During the development 
period, a variety of user studies were conducted with children, parents and educators to ensure that the materials 
represent best practices in the field. This paper summarizes the user studies and initial findings and highlights 
important issues in software design for children. 

Related Work: Previous Applications with EPSS Tools with Children 

Computer-based training and support mechanisms are an innovative approach for helping children gain 
control over personal behaviors. Although there are limited data on the use of computer-based instruction to 
support behavior change in children to date, research results are promising, Fitzgerald and Werner (1996) 
reported success with a computerized verbal mediation essay as a cognitive retraining procedure to assist a 
student with significant behavioral disorders in changing Hs behavior; the computerized essay provided 
consistent practice and focused the child’s attention and thoughts on behavioral choices and consequences. In 
another case study, the same researchers reported a procedure in which software templates were developed for a 
student to create self-monitoring materials. 

Following these early studies, the developers created a series of EPSS tools for children that supported 
children in using self-management skills in classrooms (Fitzgerald & Semrau, 1998-2000), The children’s 
software programs, First Step KidTools for ages 7-10, and Second Step KidTools for ages 11-14, provide a 
series of progressively complex templates to be used as tools. In using the tools, children identify behaviors for 
self-improvement, identify specific self-control strategies they want to use, prepare self-talk statements to guide 
their use of the strategies, and create/print-out materials to support their plans. Both software programs consist 
of tool templates that are kid-friendly with colorful graphics, text-with-audio directions, multiple examples, 
simple formats, and automatic record-keeping capabilities. To use the tools, the child simply clicks on “hot 
words” on the template form to enter personalized content and then print out the completed form for use in the 
classroom or home. 

Qualitative methodologies were used to gather multiple data on the usefulness of the approach and 
important design features in the software. Several important design decisions were made based on these studies 
that have been applied to the production of the new strategy software: (1) include the option of spoken text for 
poor readers; (2) use children as on-screen guides through the tools; (3) use children’s voices for narration; (4) 
provide access to help screens that can be accessed while using the tools; (5) provide multiple options to re-start 
or make changes; and (6) put the child in charge of creating his/her tools using natural language. Pilot testing 
with KidTools in two states demonstrated that children were able to independently use the tools and were 
motivated by the computerized tool format. Teachers reported improved motivation and self-control in 
classrooms. The findings from KidTools are extended to the production of KidSkills with its new focus on 
learning strategies. The previous work ensures appropriate literacy and ease-of-navigation features are included 
in the KidSkills design (Fitzgerald, Watson, Lynch & Semrau, 1999). 

Conceptual Framework for the Design of KidSkills 

Recognizing the importance of ecological variables surrounding an innovation systems (Peled, Peled, & 
Alexander, 1994), multiple products are being developed in a systems approach. The primary components of 
this framework are illustrated in Figure 1: development of the EPSS software, development of a resource 
information database for educators and parents, development of an orientation module, and use of online focus 
groups and a web-based hotline to share information, provide support, and disseminate materials during 
implementation. 
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KidSkills, the children’s software, offers strategies based on cognitive-behavioral approaches that help 
youngsters change cognitions (thoughts, beliefs, self-talk, cues) and behaviors (actions) within a problem- 
solving framework to gain successful outcomes. These approaches have increased in popularity during the last 
two decades as researchers have documented their effectiveness (Gresham, 1985) and curriculum developers 
have designed and disseminated teachable programs for implementation in schools (Anderson, 1981; Nichols, 
1999). To date, there have been no known precursors to this work that combines electronic performance support 
tools along with strategy training for teaching cognitive-behavioral strategies to young children. 




Fig. 1 Conceptual Framework for Development of 

The KidSkills Prototype 

There are two levels of KidSkills, First Step KidSkills is designed for ages 7-10 and Second Step KidSkills 
for ages 11-14. There are 18 tools in First Step KidSkills organized in the categories of: Getting Organized, 
Learning New Stuff, Doing Homework, and Doing Projects. There are 30 tools in Second Step KidSkills 
organized in the categories of; Getting Organized, Learning New Stuff, Organizing Information, Preparing for 
Tests, Doing Homework, and Doing Projects. Information on these strategies is provided in a resource 
information database program. Skill Resources, to provide educators and parents resources to assist children 
during implementation. A teacher orientation module is currently being developed, including a PowerPoint 
presentation, demonstration videos, and a web-based hotline for assistance. 

The software contains colorful graphics, text-with-audio directions, and simple formats. Graphic characters 
serve as “guides” to the different tools and provide audio directions in children’s voices. The text and audio in 
KidSkills allow for children to enter text in their own words. The audio directions supplement the simplified text 
instructions and can be turned on or off. To use the tools, the child simply clicks on “hot words” on the template 
form to enter his or her content and then prints the completed form for use in the classroom or home. Children 
have instant access to examples with directions and flexible ways to re-start when desired. The program 
automatically enters the child’s name, date, and establishes an audit trail for record-keeping purposes. These 
audit trails are instrumental in tracking the development of children’s thinking and skill decisions. 

Figure 2 displays the main menu screen for Second Step KidSkills, Children select tools from categories by 
clicking on the hot areas on the screen or by using the pull-down menu. Figure 3 shows an example of a 
completed Second Step tool. In this example, the KWL (Kiows. ..Wants to know... Learned) tool provides a 
learning strategy to help the learner focus on important information when learning new material and integrate 
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new information with his/her current information. This helps the learner re-structure knowledge. The child 
would enter the following information: the class, topic, source of information, what the child knows, wants to 
learn, and what was learned. The tool could be used prior to studying to guide learning, or as a follow-up to 
review information. 




Figure 2. Main Menu from Second Step KidSkills Figure 3. Sample KWLTool 

User Studies in the Developmental Process 

The software development process is recursive, going through several phases of development^ testing-^ 
revision based on procedures recommended by designers of children’s software (Druin, 1999) and evaluators of 
interactive learning systems (Reeves & Hedberg, 2002). The first step of formative evaluation, design testing , 
includes three processes: 1) review of content and interface design by experts in learning strategies and 
children’s software design, 2) observations of adults working with the prototypes, and 3) focus group meetings 
with parents and educators to discuss the tools and consumer training needs. The second step of formative 
evaluation, usability testing , includes three processes: 4) observations of children using a sample of the tools 
while collecting “think-aloud” transcripts, 5) examination of tool artifacts, and 6) expert reviews of the 
completed software. These six developmental steps lead to a full beta testing of the software, training, and 
support modules. 

Expert Review of Paper Prototypes 

Paper copies of all the tools were reviewed by eight content experts, including teachers, administrators, 
university personnel and a parent of a learning disabled child. Each reviewer used a structured questionnaire to 
provide feedback on screen design, content of the tools, terminobgy for children, and other literacy features. 
Some of the reviewers had prior experience using the KidTools program and were able to give suggestions for 
operability. Based on these reviews, some of structural elements and terminology of the tools were changed to 
make them more understandable by children with learning disabilities; some tools were re -conceptualized; color 
coding was added to guide entries; and two new tools were created. 

Design Testing in a Lab Setting 

An open lab time was scheduled at a statewide special education conference for educatora and parents to 
“play” with the KidSkills prototype. During this time, two graduate research assistants staffed the lab and 
observed 46 adults using the software, watching for difficulties or unusual routines. Field notes were made to 
record comments of participants, and feedback forms were collected. Overall ratings were 8 to 9 on a 9-point 
Likert scale (9 being the highest). Positive comments were that the program would be fun and easy to use, the 
strategies would be useful for students with learning disabilities, the audio directions were helpful, and they 
appreciated the interactivity and multisensory approaches. Some potential problems identified were difficulty in 
using pull-down menus for children, desire for more graphics and larger hot spots, use of tab key to save text 



5 best COPY AVAILABLE 



entries, desire for spell checker and thesaurus, and navigation back and forth between screens. Problems that 
were identified for implementation included teacher training, scheduling use within the school day, access to 
computers at home and school, and need for quick instructional resource. 

Online Focus Group with Consumer Group Members 

An online focus group was held with members of the target consumer groups, including one parent, three 
classroom teachers, three teacher-trainers, and a high school student with learning disabilities. Project staff also 
participated in the discussions. Four open-ended questions were used to engage participants: (1) acceptance and 
use of the software, (2) potential implementation problems, (3) recommendations for training and support, and 
(4) other suggestions. Discussion threads emerged within these broad areas. The messages were coded for 
themes using the qualitative analysis program, NVivo, With this text analysis program, information can be 
linked and examined across multiple sources, leading to an integration of the results (Richards, 1999), Some of 
the concerns that were raised were earmarked for observation of tool usage with children, particularly use of the 
tab key, size of text entry fields, quality of audio narration, and navigation problems. Discussion of 
implementation issues were helpful in designing the orientation module and planning the web hotline, 

Think-Aloud Observations of Children Using Working Prototypes 

Twelve children with mild-moderate disabilities were selected to participate in usability testing. These 
students all had learning disabilities or behavioral disorders, grades levels 27, ages 8-14, and received 
academic programming in resource or inclusive classrooms. The selected children used the software in a one- 
on-one setting with project staff After an initial training period with one of software tools, children were asked 
to "think aloud" as they worked with the software to produce materials using three different tools (Smith & 
Wedman, 1988; Someren, Barnard, & Jacobijin, 1994). An observer audiotaped the sessions and made field 
notes in order to record the children’s reactions to the interface and the overall operability of the software using 
observation and interview guidelines. These results were analyzed using 

Examination of Artifacts Created by Children Using Working Prototypes 

The artifacts produced by the children were examined to answer any questions raised during earlier stages 
of review or suggestions made about the tools during the Online Focus Group, The software contains a 
complete audit trail that records out to files all tool choices and entries made during usage. 

Expert Reviews of Completed Software 

The completed software tool programs will be sent to a panel of experts in July, 2002. The reviewers will 
have background in special education (cognitive-behavioral interventions and learning strategies) and two of the 
reviewers will have expertise in human-interface design of software (EPSS architecture and design for 
children). A systematic interface design will be conducted using an assessment guidelines based on procedures 
for user interface studies (Schneiderman, 1998), The guidelines include overall reactions, screens, sound, 
graphics, content, terminology, and operability. Reviewers will submit a written report with recommendations. 

Putting the Pieces Together 

Although each formative evaluation procedure yielded valuable information and useful suggestions, we 
found summation and integration of the information to be most critical. As discussed by Maslowski & 
Visschler, all the possible dimensions of formative and summative evaluation are rarely conducted due to 
financial and time constraints (1999). Their suggestion is to evaluate the dimensions where designers face 
questions for the greatest benefit. It is easy to respond to each procedure with revisions and further testing, as 
recommended in the recursive testing process, yet this approach can lead one down false paths based on partial, 
rather than complete, findings. For example, adults viewed the tools and raised concerns about navigation and 
the use of the tab key, yet observers of children found that with very little guidance, children quickly learned to 
navigate and enter information into tools. Adults questioned the clarity of the children’s voices for narration, yet 
observers reported that children found the voices appealing. Adults who tested the software found it easy to use, 
yet our high school student strongly recommended a hotline for teachers to answer computer-use questions. 
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It was clear that conflicting feedback was provided by different “voices” of expertise. The most useful 
dimensions to us were the direct observations of children using the tools and the messages to the Online Focus 
Group provided by our high school student with learning disabilities. He grasped how the tools could be used in 
classroom settings and the abilities and limitations of teachers in supporting roles. Above all, we learned to 
integrate the voices of children during the design process (Druin, 1999). 
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